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abstract 



Synaptic connections are known to change dynamically. High-frequency presy- 
naptic inputs induce decrease of synaptic weights. This process is known as 
short-term synaptic depression. The synaptic depression controls a gain for presy- 
naptic inputs. However, it remains a controversial issue what are functional roles 
of this gain control. We propose a new hypothesis that one of the functional 
roles is to enlarge basins of attraction. To verify this hypothesis, we employ a 
binary discrete-time associative memory model which consists of excitatory and 
inhibitory neurons. It is known that the excitatory-inhibitory balance controls 
an overall activity of the network. The synaptic depression might incorporate an 
activity control mechanism. Using a mean-field theory and computer simulations, 
we find that the basins of attraction are enlarged whereas the storage capacity 
does not change. Furthermore, the excitatory-inhibitory balance and the synaptic 
depression work cooperatively. This result suggests that the synaptic depression 
works to improve an error-correcting ability in cortical circuits. 
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1 Introduction 



Cortical neurons receive thousands of synaptic inputs. The neurons might receive 
high-frequency presynaptic inputs. Extracting intrinsic signals from random fluc- 
tuations in high-frequency presynaptic inputs is a severe challenge. Some mecha- 
nisms that reduce a gain for presynaptic inputs might work at synaptic sites. Such 
gain control cannot be achieved through fixed synaptic weights because input rates 
change dynamically. Neurophysiological experir nents show that high-frequenc y 



inputs induce the decrease of synaptic weights (|Thomson and Deuchars 



19M). 



This process is known as short-term synaptic depression. 



sion is known to control the gain for presynaptic inputs ([Abbott et al 



Tsodvks and Markram 



he synaptic depres- 



1997 



19971). This property might influence not only 



the ac- 



tivity of a single neuron but also that o. 



Kistler and Hemmen 



I999I 



Tsodvks et al 



the overall activity (jBresslofi 



20001). 



1999; 



However, it is still a controversial issue what are functional roles of the gain 
control. To elucidate the functional roles, some information must be embedded 
in the synaptic connections. We employ an associative memory model that stores 
memory patterns in synaptic connections. Only a few works have investigated how 
the synaptic depression affects the per f ormance of the associative memory model 



(Bibitchkov et al. 



2002 



Pantic et al 



patterns emb edded by Hebb rule (jHebb, 



(Amit 



2002; 



Torres et al. 



20021 ). The memory 



19491 ) become fixed points, i.e., attractors 



19891) . Bi bitchkov et al. 



storage capacity (|Bibitchkov et al, 



bund that the synaptic depression reduced a 



20021 ). Torres et al. found that the storage 



capacity decreased with the degree of the depression in a thermodynamical limit 
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( Torres et al, 



20021 ) ■ However, the main targets of these works were the steady 
states of the models. It is necessary to investigate the dynamical properties of the 
model because the synapses change dynamically. One of the important dynamical 
properties is basins of attraction which express the regions where the system 
converges to stored patterns. In the view of information processing, the basins 
of attraction reflect an error-correcting ability. Bibitchkov et al. found that 
the synaptic depression shrunk the basins of attrac tion at a large loading rate 



and increased them a little at a small loading rate ((Bibitchkov et al, 



20021) . In 



contrast, we propose a new hypothesis that the synaptic depression enlarges the 
basins of attraction not only at a small loading rate but at a large loading rate. 

We employ a binary discrete-time associative memory model which consists 
of excitatory and inhibitory neurons. The memory patterns with a small firing 
rate, i.e., sparse patterns, are embedded in the synaptic connections among the 
excitatory neurons. This coding scheme is known as sparse codin g. An activit y 



control is requisite for the stable retrieval of the sparse patterns (jOkada, 



199 



The excitatory-inhibitory balance controls an overall activity. The synaptic de- 
pression might incorporate the activity control mechanism. Specifically, when an 
overall activity is high, the synaptic depression decreases the gain for presynaptic 
inputs and maintains the overall activity at a constant level. We also investi- 
gate whether the excitatory-inhibitory balance and the synaptic depression work 
cooperatively or not. 

This paper consists of seven sections. The section 2 gives the description of the 
model employed in this paper. In the section 3 mean-field equations which describe 
a steady state of the model are shown. In the sections 4 and 5 we investigate how 



the synaptic depression influences the performance of the model which consists of 
only excitatory neurons. In the section 6 we investigate the relationship between 
the synaptic depression and the excitatory-inhibitory balance. In the section 7 
we summarize the results obtained in this paper and make a discussion about the 
model. 



2 Model 



The model used in this study consi s ts of excitatory and inhibitory n eurons (|Amit et al 



1994 



Vreeswiik and Sompolinskv 



1996 



Matsumoto et al 



20051). The z-th exci- 



tatory neuron (i = 1, ■ ■ ■ , A^) is characterized by its binary state Si(t) = {0, 1} 
and discrete time t. If the excitatory neuron fires at time t, its state is = 1; 
otherwise, Sj(t) = 0. A thermodynamics limit, A^ —* oo, is considered. The exci- 
tatory neurons are all-to-all connected. The synaptic weight from the presynaptic 
excitatory neuron j to the postsynaptic excitatory neuron i, Jij{t), are dynami- 
cally changed, and its specific value will be discussed later. The synaptic weights 
between the excitatory neurons and the inhibitory neurons are uniform. There 
are no connections among the inhibitory neurons. Therefore, the population of 
the inhibitory neurons can be regarded as a single inhibitory neuron. The state 



5 



of the i-th excitatory neuron is updated by the synchronous rule: 

s,{t + l) = e(hi{t)-g{s{t)-f)-9) (1) 

N 

= e{YlMt)sAt)-9im-f)-0) (2) 

N 

= e(X;^^.W5.W-y/W-^) (3) 

N 

= e(^j,,(t)s,(t)-e-(t)) (4) 

where hi{t) — ^j^^j (^)sj(t) denotes an internal potential, s{t) = j^J2f=i^ji^) 
denotes a mean firing rate of the excitatory neurons, / denotes a firing rate of 
memory patterns, g denotes the strength of the inhibition, ^ is a uniform threshold, 
and 9{t) = gl{t) + ^ is an effective threshold. The inhibitory neuron receives the 
mean output of the excitatory neurons, s{t) , and it sends output to the excitatory 
neurons as I{t) = s{t) — f. g = means that the model consists of only excitatory 
neurons. If the mean firing rate of the excitatory neurons, s{t), is higher than /, 
the inhibition increases an effective threshold 0{t). Then the excitatory neurons 
tend to be silent. On the contrary, if s{t) is lower than /, the inhibition decreases 
the effective threshold 6{t). Then the excitatory neurons tend to activate. The 
output function 0(-) of the excitatory neurons is a step function: 



e{u) = { (5) 




Memory patterns (/i = 1,2,--- ,p) are stored in the synaptic connections 
among the excitatory neurons. Each element of the memory pattern £^ = 
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(^1 ) '^2 5 ■ ■ ■ 5 ^n) is generated independently by 



Prob[er = 1] = 1 - Prob[ef = 0] = /. 



(6) 



The expectation of is E[^j^] = /. We consider a small firing rate /. A value 
a = p/N is defined as a loading rate. When the loading rate a is larger than a 
critical value ac, the retrieval of memory patterns become unstable. The critical 
value ac is known storage capacity. 



The closeness between s{t) and at time t is characterized by an overlap 



N 



(7) 



Nf{l - f) 

If the overlap is close to 1, i.e., m^{t) ^ 1, the model succeeds to retrieve the 
memory pattern Hereafter, the target pattern for the retrieval is the first 
memory pattern 

The synaptic weight Jij{t) incorporati ng the svnaptic depression is determined 



by a 



phenomenological model of synapses (jAbbott et al 



1997 



Tsodvks and Markram 



19971 ). When the synapses transmit input signals, they exhaust a finite amount of 
resources, e.g., neuromodulators. A dynamical amplitude factor Xj{t) denotes the 
fraction of available resources. After each spike is emitted, the resources decrease 
by a certain fraction Use and to recover with a time constant r. If the recover 
lags behind the interval of high-frequency presynaptic inputs, the amount of re- 
sources decreases and t he synapses are depressed. The factor xAt) is upd ated by 



the synaptic dynamics (|Tsodvks and Markram 



1997 



Pantic et al. 



20021 ): 



X.j{t + l) = Xj{t) + 



Xj{t) 



UsEXj{t)Sj{t). 



(8) 



Xj{t) = 1 implies that the synapses are not depressed. The synaptic weight Jij(t) 
incorporating the synaptic depression is obtained by multiplying a fixed synaptic 



weight Jij and the dynamic amphtude factor Xj{t) (0 < Xj{t) < 1): 

Mt) = J^Mt). (9) 

The synaptic weight Jij is determined by a Hebbian-hke rule, i.e., a covariance 
rule: 

4 = ^^('_^) i:(e-/)(e;-/)- (10) 

A self- connect ion Ju is assumed to be nonexistent. For simplicity, the synap- 
tic depression is incorporated into only excitatory-excitatory connections. The 
excitatory-inhibitory connections are fixed. 



3 Mean-field Equations at a Steady State 



In this section, we derive mean-field equations at a steady state, i.e., t oo. 



For simplicity, individual values at the steady state are written by Xj{oo) 



X 



3-> 



hi{oo) = hi, Si(oo) = Si, and m'^{oo) = m^. The factor a: i- ft) reaches its steady- 



state value by t — ^ oo in equation (jHJ (|Bibitchkov et al 



20021) : 



'ID 



1 + 

where 7 = tUse- The value 7 indicates the level of the synaptic depression. Since 
Sj takes a binary value, i.e., Sj = {0, 1}, the value xjSj can be written as 

Si 1 



XjSj 



-Sj. 



(12) 



l + ^Sj 1-^7 

By using this relationship, the internal potential hi at the steady state is written 



as 



N 

hi = JijXjSj 



N ^ N 

^ ' 1 + 7 1 + 7 ^ ^ 



(13) 



Then, the neuronal state Si is written as 



Si = Q{hi - g{s - /) - 



e 



1 ^ 



s,-g{s-f)-9). (14) 



By using equations (7), (9), and (12), the internal potential hi is represented as 

N p N 

E = Thmhi) E E(er - mf - /)^, 

AT 



1 r 1 



(15) 



+ 7iiV/(l-/)^ 



Nf{l-f) 



p N 1 ^ "1 

E E(^f - /)(^; - /)^^- - Nf(i-f) ^^^^ - f^''^ 

tJ.=2 j = l ■' ^ ' n=l ^ 



(e- - fW + 



1 + 7 
p 

Zi = Y.{i^ - f)m^ - asi 

fj,=2 



(16) 
(17) 



The first term of the equation (17) is a signal term for the retrieval of the target 
pattern The second term is a cross-talk noise term which represents contribu- 
tions from non-target patterns and preven ts the target pattern £^ from being re - 



trieved. According to a mean- field theory (jOkada , 



1996 



Shiino and Fukai 



19921 ) 



the cross-talk noise obeys a Gaussian distribution with mean Tsi and variance a^. 
By using this theory, the neuronal state is written as 



e 



(19) 



By applying Maxwell rule to equation (19), the solution Si is obtained as 



(20) 
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The mean-field equations describing the steady state of the model are obtained 
by the following equations. For simplicity, the overlap is written as m. 



m 
U 

Q 



1 1 

--erf(</)i) + -erf(<^2), 



/ 



27ia 



exp( 



f 



2'Ka 



exp( 



If 1 - / 

_ _ -erf(0i) ^erf(02), 

If 1 - / 

_ _ -erf(0i) ^erf(02), 



where 0i 



-(l-/)r7i+(l+7)g(§-/)+(l+7)e-£ 
V2Qcr2 : 



/m+(l+7)g(g-/)+(l+7)e-£ ^2 



0" 



(21) 
(22) 

(23) 
(24) 

(i-r/)2' 



r = Y^j, Gri{y) = eyip{—u'^)du. Solving these equations numerically, we dis- 

cuss the macroscopic state of the model. The detailed derivation of the mean-field 
equations is shown in appendix. 

The mean-field equ ations derived in t his paper are different fro m the equations 



in the previous works (IBibitchkov et al 



2002; 



Torres et al. 



20021 ). The equations 



that Bibitchkov et al. derived dropped out the equation of U (equation (22) in this 
paper). Therefore, the cross-talk noise was not estimated accurately. Torres et 
al. assumed that xj was independent of sj. However, the equation (jH)) apparently 
shows that Xj depends on Sj. Therefore, this assumption was invalid. Thus, the 
mean-field equations derived in this paper can describe the steady state of the 
model more accurately than in the previous works. 



4 Storage Capacity 

We investigate how the synaptic depression influences the steady state of the 
model. In this section, we consider excitatory neurons, i.e., g = 0. Since the 
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output function in equation (P) is a step function, the neuronal state is determined 
by the sign of an argument. Then, the neuronal state at the steady state is written 



as 



/ I ^ .AT 



'(rp: E J^^'^ - ^) = ® (tt^( E J^^'^ - (1 + ^)^)) (25) 

N 

= 0(5^J.,s,-(l + 7)^). (26) 

If the threshold 9 is set at 9 = 9/{l + -y), the neuronal state at the steady state is 
written as 

TV 

©(E^^^-^^-^)' (27) 



TV 

Si ^ 

where ^ is a threshold when the synaptic depression is not incorporated into the 



model. This equation implies that the steady state of the model incorporating the 
synaptic depression is equivalent to the one without the synaptic depression when 
the threshold is set at 9 = 9/(1 + ■y). Hereafter, the threshold with the synaptic 
depression is written as 9 while the threshold without the synaptic depression as 

e. 

Here, we show the mechanisms where the memory pattern is retrieved in 
the model with the synaptic depression. From the equation (20), the internal 
potential at the steady state is given by 



(2^ 



At first, we consider the case where the number of memory patterns is small, i.e., 
p ~ 0(1) and the synapses are not depressed, i.e., 7 = 0. The cross-talk noise does 
not exist in this case, and the second and third terms in the equation (28) vanish 
because these terms come from the cross-talk noise. Let the neuronal state at the 

11 



steady state be equivalent to the first memory pattern: s = The probabihty 
distribution of hi is shown in Figure ^a). When the threshold 6 is set between 
— / and 1 — /, the neuronal state Si takes 1 with probability / and with 1 — /. 
The retrieval of the memory pattern is successful. 

Next, we consider the case where the number of memory patterns is order A^, 
i.e., p ~ 0{N) and the synapses are not depressed, i.e., 7 = 0. The probability 
distribution of hi is shown in FigurelU^b). S etting the threshold 9 at an appropriate 



value is essential for the stable retrieval (|Qkada . 



1996 



Matsumoto and Okada . 



20021 ). Finally, we consider the case where the number of memory patterns is 
order N, i.e., p ~ 0{N) and the synapses are depressed, i.e., 7 > 0. Since 
the signal and the variance of the cross-talk noise are scaled by 1/(1 + 7) (see 
equation (28)), the probability distribution of hi is shown in Figure ^c). When 
the threshold 9 is set at 9 = 9 /{I +7), the retrieval of the memory pattern 
succeeds. 

Solving the mean-field equations (equations (21-24)) numerically, the steady 
state of the model can be analyzed. Figure Efa) shows the dependency of the 
overlap m on the loading rate a without the synaptic depression, i.e., 7 = 
at / = 0.1. The dashed lines are obtained by solving the mean- field equations 
(equations (21-24)) numerically while the error bars indicate medians and quartile 
deviations of m(lOO) obtained by computer simulations in 11 trials at = 5000. 
The initial state is set at the first memory pattern, i.e., s(0) = and the 
threshold is fixed at 6' = 0.51 which is optimized to maximize the storage capacity. 
The storage capacity ac is 0.44. Figure |2fb) shows the dependency of the overlap 
m on the loading rate a with the synaptic depression, i.e., 7 = 1.0, r = 2.0, 
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Use = 0.5, Xj{0) = 0.5, and 9 = = 0.255. The factor Xj{t) obeys equation (jHJ 
in computer simulations. The storage capacity ac is 0.44. This results show that 
the synaptic depression does not change the steady states. 



5 Basins of Attraction 

We investigate how the synaptic depression influences basins of attraction. When 
a loading rate a is less tha n a storage capacity ac, a critical overlap mc exists 



( Amari and Maginu 



19881 ). When an initial overlap m^(0) is larger than the 
critical overlap mc, the retrieval of the memory pattern succeeds. In other 
words, the system converges to the pattern Therefore, the region of m^(0) > 
mc is known as the basins of attraction. The basins of attraction express an error- 
correcting ability of the model. If the basins of attraction are enlarged, it means 
that the error-correcting ability of the associative memory model is improved. In 
this section, we consider excitatory neurons, i.e., g = 0, and this is same case as 
section 4. 

Here, we investigate the basins of attraction without the synaptic depression, 
i.e., 7 = 0. At first, we consider the case where the number of memory patterns is 
small, i.e., p ~ 0(1). Let us consider the probability distribution of the internal 
potential hi{0) at time t = 0. The peak of the distribution of = 1 is located at 
(1 — /)m^(0). If the peak of the distribution of Q = 1 is smaller than the threshold 
6, the states of all neurons become at time t = 1 and the retrieval fails (Figure 
Olja)). Therefore, the peak is larger than the threshold 6, i.e., (1 — /)m^(0) > 6 
for the successful retrieval (Figure El^b)). When the loading rate a is small, the 
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critical overlap rric satisfies the relationship of mc = j-^j. 

Next, we consider the case where the number of memory patterns is order 
N, i.e., p ~ 0{N). If the threshold is set at a small value at t = 0, the initial 
overlap m^(0) can be a small value. If the threshold is fixed at a small value, the 
threshold crosses the distribution of Q = because of the cross-talk noise (Figure 
0Ja)). At the next time t = 1, the neuronal state whose internal potential hi{0) 
is larger than the threshold 6 becomes 1 even though the neuron codes Q = 
(shadow part of FigureEfa)). The mean firing rate of the model increases, and the 
overlap decreases. Then the distribution of Q = 1 is smaller than the threshold, 
and the retrieval fails. If the threshold increases at t = 1 to keep the mean 
firing rate at a constant level, the overlap increases at the next time t = 2. As the 
threshold increases to maintain the mean firing rate at a constant level, the overlap 
increases (Figure 4(b)). This implies that if the threshold increases gradually for 
the increase of the signal in the progress of the retrieval, even though the initial 
overlap m^(0) is a small value, the retrieval succeeds. In other wor ds, the basins 



of attraction are enlarged. T his is an activity control mechanism (|Okada . 



Matsumoto and Okada 



1996 



2002). 

Here, we investigate the basins of attraction with the synaptic depression, i.e., 
7 > 0. We consider the case where the number of memory patterns is order N, 
i.e., p ~ 0{N). We set Xj{0) = 1, which implies that the synaptic depression 
does not work at time t = 0. Let the threshold 9 be fixed at a small value. The 
distribution of the internal potential hi{0) at time t = is shown in Figure a). 
In the progress of the retrieval, the overlap increases. This is a similar 
Figure El^a). By using the synaptic depression, the internal potential is written as 
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N 

^.W = $^4a;.W5.W- (29) 

In the progress of the retrieval, Xj (t) following equation (jSI) decreases. The signal 
might be fixed because the effect of the increase of the signal can be canceled out 
by the decrease of Xj{t). At a steady state, the distribution of hi is shown in Figure 
El^b). The relative relationship between the fixed threshold and the signal does 
not change in the progress of the retrieval. In the activity control mechanism, the 
relative relationship between the threshold and the signal does not change in the 
progress of the retrieval. Thus, the synaptic depression might have qualitatively 
same mechanism as the activity control mechanism. 

In order to check the qualitative consideration, we calculate the basins of 
attraction by computer simulations. Figure Efa) shows the basins of attraction 
without the synaptic depression, i.e., 7 = 0. The region in which m^(0) is larger 
than mc represents the basin of attraction for the retrieval of the target pattern 

The value mc is obtained by setting the initial state of the network at with 
additional noise. We employ the following method to add noise. lOOj/% of the 
minority components (sj(0) = 1) are flipped, while the same number of majority 
components (sj(0) = 0) are also flipped. The initial overlap m^(0) is given as 
1 — Then the mean firing rate of the model is kept equal to the firing rate of 
the memory pattern, /. When the threshold 6 is fixed at an optimal value which 
maximize the storage capacity ac, the basins of attraction are small (— in Figure 
ini^a)). When the threshold 6 is small, the basins of attraction are enlarged, but 
the storage capacity decreases (□, x, * in Figure Efa)). Figure IHfb) shows the 
basins of attraction with the synaptic depression. As discussed in section 4, the 
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storage capacity ac takes 0.44 because the threshold is set at 6 = Since 
the optimal threshold is 6* = 0.51 without the synaptic depression, the threshold 9 
with the synaptic depression is set at 9 = 9/ (1 + •y) = 0.425 (7 = 0.2,0 in Figure 
Mh)), 9 = 0.340 (7 = 0.5, X in Figure Efb)), 9 = 0.255 (7 = 1.0, * in Figure 
infb)). As the value of 7 increases, the basins of attraction are more enlarged. 

To check the qualitative consideration that the synaptic depression might have 
same mechanism as the activity control mechanism, we compare the temporal 
change of the overlap in the cases using the synaptic depression and the activity 
control mechanism. Figure [7|^a) and (b) shows the temporal change of the overlap 
m^(t) and the factor Xj(t), respectively. The parameters used in Figure [Tfa) and 
(b) are optimized to enlarge the basins of attraction. Figure Eta) indicates that 
the system converges to the target pattern within 4 time steps. Figure {Rh) 
indicates that the value Xj{t) converges at 5 time steps. This result indicates that 
it is crucial that the time constant of the convergence to is close to that of the 
convergence of Xj{t) in order to enlarge the basins of attraction. To compare the 
effect of the activity control mechanism with that of the synaptic depression, the 
temporal change of (t) with the activity control mechanism is shown in Figure 
Ulc). The threshold is set to maintain the mean firing rate of the model at /. 
Figure [Tfc) indicates that the system converges to within 5 time steps. The 
dynamics of the overlap is similar to when the synaptic depression is incorporated 
(figure [7|^ a)). The basin of attraction in this case is larger than when the synaptic 
depression is incorporated. The temporal change of the threshold is shown in 
figure [7|^d). The threshold converges at 5 time steps. These results support that 
the synaptic depression might have qualitively same mechanism as the activity 
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control mechanism. 

Bibitchkov et al. reported that the synaptic depression enlarged the basins 
of attraction a httle at a small loading rate but it mac e the basins of attraction 

20021 ) ■ Setting a threshold at 



shrink at a large loading rate (jBibitchkov et al, 



an appropriate value is critical in ^ terms of inc reasing the storage capacity and 



enlarging the basins of attraction (jOkada 



19961 ). However, Bibitchkov et ah did 



not set a threshold at an appropriate value. Thus, they could not separate the 
effect of the synaptic depression and that of a threshold. In contrast, we set a 
suitable threshold to avoid the effect of a threshold, and we can discuss only the 
effect of the synaptic depression. Thus, we obtain the results that the synaptic 
depression enlarges the basins of attraction not only at a small loading rate but at 
a large loading rate. The res ults obtained in t h is stu dy are qualitatively different 



from the results obtained bv (Bibitchkov et al. 



( 20021) 



6 Excitatory-Inhibitory Balanced Network 

In sections 4 and 5, we considered the model which consisted of excitatory neu- 
rons, i.e., g = 0. Here, we consider the excitatory-inhibitory balanced network, 
i.e., g > 0. It is known that inhibitory neurons regulate the overall activity of 
excitatory neurons. If the overall activity of the excitatory neurons goes up, the 
inhibitory neurons send strong inhibition to the excitatory neurons to suppress 
the overall activity of the excitatory neurons. If the overall activity of the exci- 
tatory neurons goes down, the inhibitory neurons become silent. This excitatory- 
inhibitory balanced network must play the role of an activity control in cortical 
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circuits. In other words, the inhibitory neurons control the effective threshold 
of the excitatory neurons. In section 5 we found that the synaptic depression 
might have qualitatively same mechanism as the activity control mechanism, even 
though the synaptic depression is a local phenomenon and the activity control 
is a global phenomenon. In this section, we investigate whether the excitatory- 
inhibitory balance and the synaptic depression work cooperatively or not. 

To do this, we consider the excitatory-inhibitory balanced network which does 
not incorporate the synaptic depression, i.e., 7 = 0. The state of the i-th excita- 
tory neuron is determined by 

N 

s,{t + l) = e(^J2j,.sj{t)-g{s{t)-f)-e) (30) 

N 

= e(j]4.,(t)-e"(t)), (31) 

where s{t) = jf^f^iSj{t) and 9{t) = g{s{t) — f) + 9. When the retrieval of 
memory patterns succeeds, the mean firing rate of the excitatory neurons, s{t), is 
close to /. Therefore, the effect of the inhibition disappears: 

g{s{t) - /) « 0. (32) 

This means that the storage capacity does not change. 

We investigate whether the balanced network enlarged the basins of attraction. 
At first, we consider the case where the number of memory patterns is order 1, 
i.e., p ~ 0(1). The cross-talk noise does not exist in this case. When the initial 
overlap m}{0) is smaller than no neurons fire at the next time t — 1. Even if 
the effective threshold 9{t) decreases at time t = 1, the distribution of = 1 also 
decreases because the overlap takes a very small value. Therefore, the retrieval 
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fails. In other words, the basins of attraction are enlarged only a little when p is 
a small value. 

Next, we consider the case where the number of memory patterns is order A^, 
i.e., p ~ 0{N). The distribution of an internal potential is shown in Figure |H1 
When the initial overlap m^(0) is smaller than ^^j, almost all neurons do not fire 
at the next time t = 1. However, at p ~ 0{N) each distribution becomes broader 
because of the cross-talk noise. This enables some neurons to activate at time 
t = 1. Since the mean firing rate of the excitatory neurons, s{t), is smaller than 
/, the effective threshold 6{t) decreases like Figure |H^b). In the progress of the 
retrieval, the effective threshold changes over time, following the value of s{t). If 
the model retrieves the target pattern s{t) is close to /. In other words, the 
inhibition disappears at the steady state (Figure |Hfc)). 

Here, the synaptic depression is incorporated into the excitatory-inhibitory 
balanced network, i.e., 7 > 0. For simplicity, we assume that the synaptic depres- 
sion is occurred on the synaptic sites among the excitatory neurons. Then the 
state of the i-th excitatory neuron can be written as 

N 

s,{t + 1) = 4x,(t)s,(t) -g{s{t) -f)-e). (33) 

We consider the case where the number of memory patterns is order 1, i.e., 
p ~ 0(1). The retrieval fails at an initial overlap m^(0) without the synaptic 
depression when (1 — /)m^(0) is lower than the threshold 9 (Figure IHl^a)). On 
the other hand, the retrieval succeeds at the same initial overlap m^(0) with the 
synaptic depression (FigureE^b)) because (1 — /)m^(0) is higher than the thresh- 
old 6 = j^. Therefore, the basins of attraction are enlarged at a small loading 
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rate. At a large loading rate {p ~ 0{N)), the effect of the synaptic depression is 
smaller than that of the inhibition because of the cross-talk noise. Thus, at a large 
loading rate, the synaptic depression does not change the basins of attraction in 
the excitatory-inhibitory balanced network. 

In order to check the qualitative consideration, we calculate the basins of at- 
traction by computer simulations. Figure ITUT a) shows the basins of attraction 
without the synaptic depression. As the value of g increases, the basins of attrac- 
tion are enlarged. At a small loading rate the basins are enlarged a little. Thus, 
the excitatory- inhibitory balanced network enlarges the basins of attraction. At 
a small loading rate it enlarges the basins a little. Figure [TUT b) shows the basins 
of attraction with the synaptic depression. To compare the effect of the synaptic 
depression, the case of the balanced network without the synaptic depression (□) 
is shown. In the balanced network with the synaptic depression, the basins of 
attraction are the largest (*). Even at a small loading rate the basins are en- 
larged. Thus, the excitatory-inhibitory balance and the synaptic depression work 
cooperatively. 

7 Discussion 

In this paper, we investigated how the synaptic depression influenced the perfor- 
mance of the associative memory model in terms of the storage capacity and the 
basins of attraction. Using the mean-field theory and the computer simulations, 
we found that the basins of attraction were enlarged whereas the storage capac- 
ity did not decrease. In other words, the synaptic depression had the mechanisms 
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that the neuron threshold effectively increased in the progress of the retrieval. Fur- 
thermore, the excitatory-inhibitory balance and the synaptic depression worked 
cooperatively in the excitatory-inhibitory balanced network. This result suggests 
that the short-term synaptic depression might improve an error-correcting ability 
in cortical circuits. 

In our model the synaptic depression was assumed to occur only at the synap- 
tic sites among excitatory neurons. In the cortical circuits the synaptic depression 
occurs not only at excitatory synaptic sites but at inhibitory synaptic sites. Re- 
cently, Galarreta and Hestrin found that long-term firing induced a much stronger 
depression of excitatory synapses than that of inhibitory synapses although the 
i nitial rates of depression were similar at both excitatory and inhibitory synapses 



( Galarreta and Hestrin 



19981 ). The result implies that the synaptic depression 
induces strong depression at excitatory synapses, it induces weak depression at 
inhibitory synapses. If the weak depression at inhibitory synapses is regarded as 
no depression in our model, our assumption that the synaptic depression occurs 
among the excitatory neurons might be valid. 
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Appendix. Derivation of Mean-Field Equations 

In this appendix, we derive mean-field equations at a steady state, i.e., t — oo. For 
simplicity, each value at the steady state is written by Xj{oo) = xj, hi{oo) = hi, 
m^(oo) = m^, and Si(o o) = Sj. The factor xAt ) reaches its steady-state value by 



t — > CX3 in equation (jSI) (jBibitchkov et al 



2002): 



1 



(34) 



' 1 + 7^/ 

where 7 = tUse- Since Sj takes binary values, i.e., Sj = {0, 1}, written as 



1 



XjSj 



-Sj. 



l + 'ysj 1 + 7 

By using this relationship, the state of i-th excitatory neuron is written as 



(35) 



JV 



e 



e 



N 



N 



( E J^i'^ - (1 + 7)^?(^" -/)-(!+ 7)^ 



(36) 
(37) 
(38) 
(39) 



where Jij = J2^fi=i i^i ~ f) i^j ^ /) ^ indicates a threshold when the 

synaptic depression is incorporated into the model. The neuronal state can be 
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transformed by 

p N 

.Nf(l - /) ^ 



(40) 

P N 



^^TT - fm - f)s^ - (1 + l)9{s -/)-(! + 7)^^) (41) 



= e{i2(^i'-f)m''-as,-{l + ^)g{-s-f)-{l + ^)e). (42) 

We assume that equation can be solved by using an effective response function 
F(-) as 

'^ = ^ ( E(C - f)m^ - (1 + l)9is -/)-(! + 7)^) • (43) 

Let be a target pattern to be retrieved. We assume that is order of 1 with 
respect to N (m^ ~ and (/x > 1) is order of with respect to N 

{m^ ~ 0(^)). We apply Taylor expansion to obtain 

p 



= F((er-/)m'' + J](er-/)m^- (1 + 7)^7(5- -/)-(! +7)^') (44) 

= F ( - f)"^^ - (1 + ^)3{s -/)-(! + 7)^") 

p 

+ (e - f)m>^F' ( ^(er - /)m'^ - (1 + j)g{s -/)-(!+ 7)^) (45) 



= s[^'> + {C~f)m^s't\ (46) 

where 4^^^ = ^(^".^^(a - /)"^" - (1 + 7)^7(s- -/)-(! + 7)^) and s'}"^ = 
F' ( E!:^;.(er - f)^" - (1 + 7)^7(^ - /) - (1 + 7)^) • By using this relationship, 
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an overlap is obtained as 

N 

= T7J7^^^T.(i^-f>^ (47) 
1=1 

N , N 



1 


Nf{l- 


f) 


1 




Nf{l- 


f) 


1 




Nf{l- 


f) 



N 



(49) 



where U — ^ Z^ili ^'i'^^- The overlap is obtained as 



1 1 ^ 



JV/(1-/) 



i=l 

The first term of equation (39) is transformed by 

N ^ P N 

- i^' - f)^" + Nfil-f)l-U £ ^^^^ - ^^^^^ - ^^''^ - ^'^^ 



M=2 i=l 
p AT 



(54) 

= (e- - /)^' + 5i + Tsi (55) 

where ~z, = ^vTcW) E^=2 E,y.(er - ' f)sf and T = ^. In this 

transformation we used Sj — s^f^ because of Si — s^f' ~ 0{^). The first term 
in equation (55) is a signal term for the retrieval of the target pattern while 
the second term is a cross-talk noise. Since is independent of ^j*, Zi obeys a 
Gaussian distribution whose average is and variance is cr^, i.e. Zi ~ A^(0, cr^). 
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Since the square of zi is calculated by 

1 1 P ^ 2 

the variance cr^ is calculated by 

a'^mi)'\-{m\f (57) 

where q — -^Yl!j=i{^'^j^^Y ■ The third term in equation (55) is an average term 
of the cross-talk noise and denotes an effective self-coupling term. Since Sj is 
symmetric about i, Sj can be replaced by Y: 

Y = - fW + 5i + rr - (1 + ^)g{s -/)-(! + 7)^) ■ (58) 

By using a random variable Zi ~ A'^(0, 1), y is written as 

Y^Q [{il - fW + + ry - (1 + ^)g{s -/)-(! + 7)^^) ■ (59) 

We assume a self-averaging property to hold so that a site average can be replaced 
by an average over the memory patterns and random variable 2;^ ~ iV(0, 1) in which 
the random variation of Zi from site to site can be described by a single noise z 
obeying an identical Gaussian distribution. After rewriting — > ^, — > m, and 
Si — > y, and using the relationship of Sj = s^f' because of Sj — s^f^ ~ 0(-^), the 
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/(I-/) 

J Dz «Y^ », (61) 
U ^ ^ I Dzz « Y », (62) 



order-parameter equations are obtained as 

m = -—^-—[Dz«{^-f)Y», (60) 



Dz « Y », (63) 
y = e[{C-f)m + az + rY-{l + ^)g{s-f)-il + ^)9y (64) 

- - ^TTuy- (65) 

where << • • • >> denotes the average over the memory pattern ^ and random 
variable z with the probabihty density Dz = ^^exp(— ^). In general, the equa- 
tion of Y (equation (64)) admits more then one solution. In that case, Maxwell 
rule should be applied to choose the appropriate solution. The solution Y is given 

by 

Y^e(^{^-f)m + az + ^-{l + ^)g{s -/)-(! + 7)^) ■ (66) 
Finally, we get the mean-field equations. 

^ = 7(137) /^^«(^-/)^» (67) 

- 7(T^;i/exp(4)«(^-/)K(^-^)- + - + ^ 

-(l-h7)5(^- /)-(! + 7)^) » (68) 

= -lerf(0i) + ^erf(02), (69) 



29 



where erf(y) = ^ /^xp(-«^)rf«, 0, = ^ ^ 

/m+(l+7)9(s-/)+(l+7)e-i 



\/2aa 



.2 



= J Dz «Y'^ » (70) 

^ vb / ^""P^-y ^ {® (^^ - /)m + + ^ - (1 + 7)^(«- -/)-(! + 7)^) }' » 

(71) 

= ^ - {erf(0O - ^^erf(02). (72) 



= ^ J Dzz «Y » (73) 

= J ;,exp(-^)«e((e-/)m + a^ + ^-(l + 7)^(s -/)-(! + 7)^) » 

(74) 



exp(-0^) + exp(-0^), (75) 



s = I Dz«Y » (76) 
^ y exp(-^) « e (({ - /)m + a^; + ^ - (1 + 7)<7(s -/)-(! + 7)^) » 



(77) 

= ^ - {erf(0i) - ^^erf(02). (78) 
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Figure captions 



Figure 1: The probability distribution of hi at the steady state. (a):p ~ and 
the synapses are not depressed. {h):p ~ 0{N) and the synapses are not depressed. 
(c):p ~ 0{N) and the synapses are depressed. 

Figure 2: The dependency of m on a loading rate a at / = 0.1. The dashed 
lines are obtained by solving the mean-field equations numerically while the error 
bars indicate medians and quartile deviations of m(lOO) obtained by computer 
simulations in 11 trials a,t N — 5000. (a):the case without the synaptic depression, 
i.e., 7 = 0, and 9 = 0.51. ac = 0.44. (b):the case with the synaptic depression, 
i.e., 7 = 1.0, T = 2.0, Use = 0.5, Xj{0) = 0.5, and 9 = 0.255. ac = 0.44. 

Figure 3: The probability distribution of /ij(0). p ~ 0(1) and the synapses are 
not depressed. (a):the peak of the distribution of = 1 is smaller than the 
threshold 9. (b):the peak of the distribution of Q = 1 is larger than the threshold 
9. 

Figure 4: The probability distribution of the internal potential, p ~ 0{N) and 
the synapses are not depressed. (a):the threshold 9 is fixed at a small value. 

(b):the threshold 9{t) increases for the increase of the signal. 

Figure 5: The probabihty distribution of the internal potential, p ~ 0{N) and 
the synapses are depressed. (a):at time t = 0. (b):at the steady state. 
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Figure 6: The dependency of mc on a at / = 0.1. The dashed hnes are obtained 
by solving the mean-field equations (same solid line in Figure 2) while the error 
bars indicate medians and quartile deviations obtained by computer simulations 
in 11 trials at = 5000. (a):the synapses are not depressed and —:9 — 0.51, 
n-.O — 0.425, x:9 = 0.365 and *:9 = 0.255. (b):the synapses are depressed and 
Xj{0) = 1.0. -:§ = 0.51, 7 = 0.0. 0:0 = 0.425, 7 = 0.2, r = 1.2, and Use = 0.167. 
x:^ = 0.340, 7 = 0.5, r = 1.5, and Use = 0.333. *:§ = 0.255, 7 = 1.0, r = 2.0, 
and Use — 0.5. 

Figure 7: (a):the temporal change of m^(t) with the synaptic depression. Each 
solid line shows the temporal change of m^(i) when the initial overlap m^(0) is set 
at 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0, respectively. These lines were 
obtained by computer simulations at r = 2.5, Use — 0.2, Xj{0) — 1.0, p — 500, and 
= 5000. (b):the temporal change of x.j{t) when the initial value Xj{0) is set at 
0, and the fine is obtained by equation (8) at r = 2.5, Use — 0.2, and Xj{0) — 1.0. 
(c):the temporal change of m^(t) with the activity control mechanism. Each solid 
line shows the temporal change of m}{t) when the initial overlap m}{0) is set at 
0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0, respectively. These lines were 
obtained by computer simulations at p = 500 and N = 5000. (d):the temporal 
change of 9{t) when the initial overlap m^(0) is set at 0.2. 

Figure 8: The distributions of the internal potential when the synapses are not 
depressed. (a):at t = 0, (b):at t = 1, (c):at steady state. 
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Figure 9: The probability distribution of hi{0) at time t = and p ~ 0(1)- 
(a): the synapses are not depressed, and the peak of the distribution = 1 is 
smaller than the threshold 9. (b):the synapses are depressed, and the peak of the 
distribution = 1 is larger than the threshold 9 at Xj{0) = 1.0. 



Figure 10: The basins of attraction in the excitatory- inhibitory balanced network. 
The dashed lines are obtained by the mean-field equations while the error bars 
indicate medians and quartile deviations obtained by computer simulations in 
11 trials at N — 5000 and / = 0.1. (a): without the synaptic depression at 
9 — 0.51. —:g = 0, x:g = 1, *:g = 2, n-.g = 4.5. (b)*:with the synaptic 
depression {9 — 0.425, 7 = 0.2, g — 4.5) Diwithout the synaptic depression 
(^ = 0.51,^ = 4.5). 
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